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TITLE OF THE INVENTION 



AUGMENTED REALITY PRESENTATION APPARATUS AND METHOD, 

AND STORAGE MEDIUM 

5 

FIELD OF THE INVENTION 
The present invention relates to an augmented 
reality presentation apparatus and method, which 
superimposes a virtual object on a real space, and a 
10 storage medium. 

More specifically, the present invention relates 
tc an augmented reality presentation apparatus and 
method having a function of presenting a subjective 
augmented reality view for a player and generating a 
q 15 video of the same augmented reality space observed from 

a : 

~~ an objective viewpoint, and a storage medium. 
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BACKGROUND OF THE INVENTION 
A conventional augmented reality (AR) game system 
20 is constructed by an AR game apparatus for controlling 
an AR game, a head-mounted display (to be abbreviated 
as an HMD hereinafter) which the player of the AR game 
wears, and a display for presenting a subjective 
viewpoint video of the player to a third party other 
25 than the player. 



- 1 - 



The AR game apparatus controls the AR game, and 
generates information that pertains to rendering of a 
virtual object at the same time. The AR game apparatus 
generates a composite video as a video obtained by 
compositing a video of a virtual object generated using 
the information that pertains to rendering of the 
virtual object, and a video of a real space at the 
subjective viewpoint of the player, which is sensed by 
a camera attached to or built in the HMD. 

The composite video is displayed on the display 
screen of the HMD. The player plays the AR game while 
reviewing this composite video. In addition to the 
player, the video at the subjective viewpoint of the 
player as this composite video is presented to a third 
party who watches this AR game in addition to the 
player by displaying that video on a display prepared 
for a third party other than the player. 

Some TV programs use virtual studios in each of 
which a program stage set is built by virtual objects. 
In this virtual studio, a program video is generated by 
compositing an actually sensed video of characters of 
the program sensed by a set camera, and a video of the 
virtual studio seen at the position and posture of that 
camera . 

In the conventional AR (augmented reality) game 
system, a third party (watcher) other than the player 



can only watch a video generated for the player (mainly 
at the subjective viewpoint of the player) . For this 
reason, the watcher cannot watch videos of the AR game 
at viewpoints other than the subjective viewpoint of 
the player, e.g., the overall view of the AR game 
viewed from the objective viewpoint, a video of the AR 
game at a desired viewpoint of the watcher, and the 
like and can neither recognize the overall status of 
the AR game nor watch the AR game from a desired 
viewpoint . 

Also, a program video using a virtual studio is 
generated at the objective viewpoint of a third party 
other than the characters of the program, but a video 
viewed from the subjective viewpoint of each character 
of the program cannot be generated. 

It is, therefore, an object of the present 
invention to generate a video of an AR game at the 
subjective viewpoint of the player and also generate 
the overall view of the AR game viewed from an 
objective viewpoint and a video of the AR game viewed 
from a desired viewpoint of the watcher. 

SUMMARY OF THE INVENTION 
In order to achieve the above object, an 
augmented reality presentation apparatus described in 
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claim 1 as a preferred embodiment of the present 
invention comprises the following arrangement. 

That is, augmented reality presentation means for 
superimposing the virtual object viewed from a player's 
5 viewpoint position in the real space viewed from said 
player's viewpoint position; 

the first video sensing means for sensing a video 
of the real space viewed from a first viewpoint 
position which differ from said player' s viewpoint 
10 position; 

the first video generation means for generating a 
video of the virtual object viewed from said first 
viewpoint position; 
and 

15 the first video composition means for compositing 

an augmented reality video viewed from said first 
viewpoint position on the basis of said videos of the 
real space and the virtual object viewed from said 
first viewpoint position. 

20 As a result, a video that reviews an operating 

state of a virtual object by the player in the 
augmented reality space from the objective viewpoint 
position can be generated and presented to a third 
party other than the player. 
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The augmented reality presentation apparatus as a 
preferred embodiment of the present invention further 
comprises the following feature described in claim 2. 
That is, said augmented reality presentation 
5 means further comprises: 

the second video sensing means for sensing a 
video of the real space viewed from said player' s 
O viewpoint position; 

the second video generation means for generating 
10 a video of the virtual object viewed from said player's 
viewpoint position ; 

the second video composition means for 
compositing an augmented reality video viewed from said 
player' s viewpoint position on the basis of said videos 
15 of the real space and the virtual object viewed from 
said player's viewpoint position; 
and 

the display means for displaying to the player 
the augmented reality video viewed from said player' s 
20 viewpoint position. 

The augmented reality presentation apparatus as a 
preferred embodiment of the present invention further 
comprises the following feature described in claim 3. 
That is, said augmented reality presentation 
25 means further comprises: 
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the second video generation means for generating 
a video of the virtual object viewed from said player's 
viewpoint position; 
and 

5 the display means for displaying to the player 

the video of the virtual object viewed from said 
player' s viewpoint position on a display surface 
f« through which the player can visually see the real 

jU space. 

10 The augmented reality presentation apparatus as a 

preferred embodiment of the present invention further 
comprises the following feature described in claim 4 . 

That is, information generation means for 
generating information that pertains to rendering of 
15 the virtual object, and 

in that said first video generation means and 
said second video generation means generate videos of 
the virtual object using the information that pertains 
to rendering of the virtual object. 
20 The augmented reality presentation apparatus as a 

preferred embodiment of the present invention further 
comprises the following feature described in claim 5. 

That is, said information generation means 
generates information of an outer appearance of the 
25 virtual object and information of a position/posture of 
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the virtual object as the information that pertains to 
rendering of the virtual object. 

The augmented reality presentation apparatus as a 
preferred embodiment of the present invention further 
5 comprises the following feature described in claim 6. 

That is, parameters of said first video sensing 
means are known, and 

said first video generation means generates the 

S video of the virtual object viewed from said first 

W 

^ 10 viewpoint position in accordance with the known 

ins 

parameters. 

^ The augmented reality presentation apparatus as a 

M preferred embodiment of the present invention further 

comprises the following feature described in claim 7. 
15 That is, some of parameters of said first video 

sensing means are variable, 

said apparatus further comprises measurement 
means for measuring changes of the parameters, 
and 

20 said first video generation means generates the 

video of the virtual object viewed from said first 
viewpoint position in accordance with the parameters 
measured by said measurement means. 

When the parameters of the objective viewpoint 
25 video sensing means, the objective viewpoint video 
generation means receives parameters from the 



s 



- 7 - 



measurement means, and generates an objective viewpoint 
video according to the received parameters. 

The augmented reality presentation apparatus as a 
preferred embodiment of the present invention further 
comprises the following feature described in claim 8. 

That is, the parameters of said first video 
sensing means measured by said measurement means 
include at least one of a viewpoint position/posture, 
and zoom ratio. 

The objective viewpoint video of a virtual object 
is generated in accordance with camera parameters > 
(external parameters (viewpoint position/posture) and 
internal parameters (zoom ratio, aspect ratio, optical 
axis central position, distortion ratio) ) of the 
objective viewpoint video sensing means. The camera 
parameters measured by the measurement means preferably 
include all parameters to be changed dynamically of 
those parameters. 

Other features and advantages of the present 
invention will be apparent from the following 
description taken in conjunction with the accompanying 
drawings, in which like reference characters designate 
the same or similar parts throughout the figures 
thereof . 



BRIEF DESCRIPTION OF THE DRAWINGS 




The accompanying drawings, which are incorporated 
in and constitute a part of the specification, 
illustrate embodiments of the invention and, together 
with the description, serve to explain the principles 
of the invention. 

Fig. 1 is a view showing the first embodiment; 

Fig. 2 is a block diagram showing the flow of 
processes of the first embodiment; 

Fig. 3 is a diagram showing the generation 
process of a coordinate conversion matrix used to 
convert the object coordinate position of a virtual 
object into an image coordinate position viewed from a 
given viewpoint; 

Fig. 4 is a block diagram showing the flow of 
processes of the third embodiment; 

Fig. 5 is a view showing a video presented to a 
player in the first to fourth embodiments; 

Fig. 6 is a view showing a video presented to a 
player in the fifth embodiment; 

Fig. 7 is a block diagram showing the internal 
arrangement of an AR game apparatus; 

Fig. 8 is a flow chart for generating a 
subjective viewpoint AR video and an objective 
viewpoint AR video; 

Fig. 9 is a diagram showing a camera system used 
in the fourth embodiment; 



Fig. 10 is a flow chart in the fourth embodiment; 

Fig. 11 is a flow chart in the fifth embodiment; 

Fig. 12 is a table showing camera parameters of a 
plurality of cameras in the fourth embodiment; 

Fig. 13 is a view for explaining an optical 
see-through HMD; 

Fig. 14 is a block diagram showing the flow of 
processes of the second embodiment; 

Fig. 15 is a block diagram showing the sixth 
embodiment; and 

Fig. 16 is a block diagram showing the seventh 
embodiment . 

Fig. 17 is a block diagram showing the flow of 
processes of the tenth embodiment. 



DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Preferred embodiments of the present invention 
will now be described in detail in accordance with the 
accompanying drawings . 
[First Embodiment] 

As one preferred embodiment of the present 
invention, this embodiment will explain an augmented 
reality presentation apparatus and method, which 
present a state in which a player is playing an AR 
(augmented reality) game in an AR space to a third 



party (to be referred to as a watcher hereinafter) 
other than the player. 

Fig. 1 shows this embodiment. 

Reference numeral 101 denotes a player who is 
playing an AR (augmented reality) game, and wears a 
head-mounted display (to be abbreviated as an HMD 
hereinafter) 107 on his or her head. Note that the HMD 
107 in this embodiment is of video see-through type 
which possesses video camera (s) inside or on it to 
capture the video (s) from the same viewpoint of the 
player's eye(s). Since the video see-through HMD is 
known to those who are skilled in the art, a detailed 
description thereof will be omitted. 

Reference numeral 102 denotes virtual objects as 
characters which appear in the AR game. Each virtual 
object 102 is a three-dimensional virtual object 
rendered by computer graphics. The virtual objects 102 
are generated by a method to be described later by an 
AR game apparatus 104. The player 101 can review a 
video obtained by superimposing the virtual objects 102 
on a real space (to be referred to as an AR video 
hereinafter) from his or her subjective viewpoint by 
wearing the HMD 107 on his or her head, and can 
consequently play the AR game. 

Reference numeral 103 denotes an objective 
viewpoint video sensing camera (to be simply referred 



to as a camera hereinafter) for sensing a state in 
which the player 101 is playing the AR game. In this 
embodiment, this camera 103 is fixed at a predetermined 
position and posture. A video sensed by the camera 103 
(to be referred to as an objective viewpoint actually 
sensed video) is sent to the AR game apparatus 104 via 
a cable shown in Fig. 1. Note that the sensed video is 
an actually sensed video, which does not include any 
images of the virtual objects 102. 

Reference numeral 104 denotes an AR game 
apparatus which controls the AR game, generates the 
videos of the virtual objects 102, and generates an AR 
video to be output to the HMD 107 and a display 106. 
Note that a video which the AR game apparatus 104 
outputs to the HMD 107 is an AR video (to be referred 
to as a subjective viewpoint AR video hereinafter) 
obtained by compositing an actually sensed video of the 
real space (to be referred to as a subjective viewpoint 
actually sensed video hereinafter) input from the HMD 
107, and videos of the virtual objects 102 viewed from 
the subjective viewpoint (to be referred to as 
subjective viewpoint virtual videos hereinafter) . On 
the other hand, a video that the AR game apparatus 104 
outputs to the display 106 is an AR video (to be 
referred to as an objective viewpoint AR video 
hereinafter) obtained by compositing videos of the 



virtual objects 102 viewed from the objective viewpoint 
(to be referred to as objective viewpoint virtual 
videos hereinafter) , and the objective viewpoint 
actually sensed video sensed by the camera 103. 

Reference numeral 105 denotes a table as a real 
object used as a stage in the AR game. 

Reference numeral 106 denotes a display which 
displays the objective viewpoint AR video generated by 
the AR game apparatus 104 to present the playing state 
of the AR game by the player 101 to a third party other 
than the player 101, as described above. 

Reference numeral 107 denotes the aforementioned 
HMD which displays the subjective viewpoint AR video 
generated by the AR game apparatus 104 on its display 
screen 501 shown in Fig. 5. Also, the HMD 107 senses 
the subjective viewpoint actually sensed video viewed 
from the subjective viewpoint of the player 101. The 
sensed video is sent to the AR game apparatus 104 via a 
cable shown in Fig. 1. 

The flow of processes until the AR game apparatus 
104 displays the subjective and objective AR videos 
respectively on the HMD 107 and display 106 based on 
the aforementioned arrangement will be explained below 
using Fig. 2 that shows the flow of such processes. 

Reference numeral 201 denotes a game state 
manager incorporated in the AR game apparatus 104. The 



game state manager 201 manages the state of the AR game 
(information that pertains to rendering of the virtual 
objects 102, the score of the player 101, AR game round 
count, and the like) . Note that the information that 
pertains to rendering of the virtual objects 102 
includes information of the outer appearance of each 
virtual object 102, and information of the 
position/posture of each virtual object 102 in the 
world coordinate system. 

The information of the outer appearance of each 
virtual object 102 is information that pertains to 
polygons which build the virtual object 102, i.e., the 
number of polygons, the coordinate values of polygons, 
the colors of polygons, and the like. When each 
virtual object 102 has undergone texture mapping, the 
information of the outer appearance of each virtual 
object 102 includes a texture size, texture file name, 
and the like. 

Reference numeral 204 denotes a camera parameter 
measurement /management unit which measures/manages 
camera parameters as parameters of the HMD 107 and 
camera 103. The camera parameters to be managed by the 
camera parameter measurement /management unit 204 
includes viewpoint position/posture information as 
external parameters, and information of the field angle, 
focal length, distortion, and the like as internal 



parameters. The camera parameter measurement /management 
unit 204 stores the internal parameters of the HMD 107 
as known information. At the same time, the unit 204 
measures the external parameters (information of the 
viewpoint position and posture) of the HMD 107, and 
manages the camera parameters of the HMD 107. 
Furthermore, the camera parameter 
measurement/management unit 204 manages camera 
parameters of the camera 103 as known information. 

Reference numeral 202 denotes an objective 
viewpoint video generator built in the AR game 
apparatus 104. The objective viewpoint video generator 
202 generates objective viewpoint virtual videos on the 
basis of the information that pertains to rendering of 
the virtual objects 102, which is input from the game 
state manager 201. 

Reference numeral 203 denotes an objective 
viewpoint video composition unit built in the AR game 
apparatus 104, which generates an objective viewpoint 
AR video by compositing objective viewpoint virtual 
videos generated by the objective viewpoint video 
generator 202, and an objective viewpoint actually 
sensed video input from the camera 103. 

Generation of the subjective viewpoint video will 
be explained below. 



The game state manager 201 updates the 
information that pertains to rendering of the virtual 
objects 102 as needed, and outputs the updated 
information to a subjective viewpoint video generator 
212. The subjective viewpoint video generator 212 
generates videos of the virtual objects 102 (subjective 
viewpoint virtual video) viewed from the viewpoint 
position/posture of the HMD107, on the basis of the 
information that pertains to rendering of the virtual 
objects 102, which is input from the game state manager 
201, and the camera parameters of the HMD 107, which 
are input from the camera parameter 
measurement/management unit 204. 

The generated subjective viewpoint virtual video 
is output to a subjective viewpoint video composition 
unit 213. 

The subjective viewpoint video composition unit 
213 receives from the HMD 107 the subjective viewpoint 
actually sensed video that the player 101 watches via 
the HMD 107. The subjective viewpoint video 
composition unit 213 generates a subjective viewpoint 
AR video as a composite video of this subjective 
viewpoint actually sensed video, and the subjective 
viewpoint virtual videos input from the subjective 
viewpoint video generator 212, and outputs that 
subjective viewpoint AR video to the HMD 107. The 
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subjective viewpoint AR video is displayed on the 
display screen 501 of the HMD 107, and the player 101 
plays the AR game while reviewing this subjective 
viewpoint AR video. 
5 Generation of an objective viewpoint video will 

be explained below. 

The game state manager 201 updates the 
information that pertains to rendering of the virtual 
objects 102 as needed, and outputs the updated 
10 information to the objective viewpoint video generator 
202. 

Since generation of the videos of the virtual 
objects 102 is implemented by the same processes as 
those in generation of the subjective viewpoint video, 
15 a detailed description thereof will be omitted. 

The generated objective viewpoint virtual video 
is output to the objective viewpoint video composition 
unit 203. 

The objective viewpoint video composition unit 
20 203 receives an objective viewpoint actually sensed 
video from the camera 103. The objective viewpoint 
video composition unit 203 generates an objective 
viewpoint AR video as a composite video of this 
objective viewpoint actually sensed video, and the 
25 objective viewpoint virtual videos input from the 

objective viewpoint video generator 202, and outputs 



- 17 - 



the objective viewpoint AR video to the display 106. 
This objective viewpoint AR video is displayed on the 
display 106, and the watcher can see the overall view 
of the AR game by reviewing this objective viewpoint AR 
video and can recognize the current situation of the AR 
game . 

The image coordinate values of the vertices of 
the polygons that build each virtual object 102 viewed 
from each viewpoint (HMD 107 or camera 103) can be 
computed using the coordinate conversion matrix Ml or 
M2 from the object coordinate system into the image 
coordinate system. The generation process of such 
coordinate conversion matrix will be explained below 
using the block diagram shown in Fig. 3. 

A coordinate conversion matrix Mm from the object 
coordinate system into a world coordinate system is 
computed on the basis of the information of the 
position/posture of each virtual object 102. Also, 
coordinate conversion matrices Mcl and Mc2 from the 
world coordinate system into the corresponding camera 
coordinate systems are respectively computed on the 
basis of information of the positions/postures of 
respective viewpoints included in the camera parameters 
of the HMD 107 and camera 103. Furthermore, coordinate 
conversion matrices Mdl and Md2 for implementing 
perspective conversions from the corresponding camera 
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coordinate systems into the image coordinate system are 
respectively computed on the basis of information such 
as the field angles, focal lengths, distortions, and 
the like included in the camera parameters of the HMD 
5 107 and camera 103. The aforementioned coordinate 
conversion matrices are computed by the subjective 
viewpoint video generator 212 or objective viewpoint 
video generator 202. 

The conversion matrix Ml from the object 
10 coordinate system into the image coordinate system of 
the HMD 107 is given by: 

Ml = MdlMclMm 
On the other hand, the conversion matrix M2 from the 
P object coordinate system into the image coordinate 

q 15 system of the camera 103 is given by: 

" * M2 = Md2Mc2Mm 

These computations are made by the subjective viewpoint 
video generator 212 or objective viewpoint video 
generator 202. 
20 The present invention is not limit to these 

computations to make images. These computations are 
used, as factors that characterize subjective viewpoint 
and objective viewpoint. 

The internal arrangement and operation of the AR 
25 game apparatus 104 will be explained below using Fig. 7 
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that shows the internal block diagram of the AR game 
apparatus 104 . 

Reference numeral 701 denotes a CPU, which 
executes a program code loaded onto a RAM 703. The CPU 
701 also has an area for temporarily saving data during 
execution of a program. 

Reference numeral 702 denotes a ROM which stores 
setups upon and after starting up the AR game apparatus 
104, and a startup program code. Also, the ROM 702 
stores character codes and the like used to display on 
the display screen 501 a score or the like output to 
the HMD 107 during the AR game. 

Reference numeral 703 denotes a RAM which stores 
a program code of the AR game loaded from a floppy disk, 
CD-ROM, or the like as an external storage medium, 
polygon and texture data that build each virtual object 
102, and the like. Note that the game state manager 
201 refers to and manages polygon (texture) data that 
build each virtual object 102, which are stored in the 
RAM 703. 

Reference numeral 704 denotes an interface (to be 
abbreviated as an 1/F hereinafter) , which is used to 
connect the AR game apparatus 104 to an external 
apparatus. The HMD 107, camera 103, and display 106 
are all connected to the AR game apparatus 104 via this 
I/F 704. 



Reference numeral 705 denotes a console which 
comprises a keyboard and a pointing device such as a 
mouse or the like. This console 705 allows the user to 
input setup commands of the AR game apparatus 104, and 
those for peripheral devices connected to the AR game 
apparatus 104. 

Reference numeral 706 denotes a bus that connects 
the aforementioned units. Note that the game state 
manager 201, objective viewpoint video generator 202, 
objective viewpoint video composition unit 203, 
subjective viewpoint video generator 212, subjective 
viewpoint video composition unit 213, and camera 
parameter measurement/management unit 204 are connected 
to this bus 706, and are controlled via the bus 706 on 
the basis of the program code loaded onto the RAM 703, 
as described above. 

When the CPU 701 executes the program code loaded 
onto the RAM 703, the game state manager 201, objective 
viewpoint video generator 202, objective viewpoint 
video composition unit 203, subjective viewpoint video 
generator 212, subjective viewpoint video composition 
unit 213, and camera parameter measurement/management 
unit 204 execute the aforementioned processes, and 
output the subjective and objective viewpoint AR videos 
to the HMD 107 and display 106, respectively. The flow 



chart of this program code is shown in Fig. 8, and will 
be described below. 

In step S801, the camera parameter 
measurement /management unit 204 updates the information 
of the viewpoint position/posture of the HMD 107. 

In step S802, the game state manager 201 updates 
the game state (information that pertains to rendering 
of the virtual objects 102). 

In step S803, the objective and subjective 
viewpoint video generators 202 and 212 respectively 
generate objective and subjective viewpoint videos. 

In step S804, the objective and subjective 
viewpoint video composition units 203 and 213 
respectively receive actually sensed videos sensed by 
the HMD 107. and 103. 

In step S805, the objective and subjective 
viewpoint video composition units 203 and 213 
respectively generate objective and subjective 
viewpoint AR videos. 

In step S806, the objective and subjective 
viewpoint video composition units 203 and 213 
respectively output the objective and subjective 
viewpoint AR videos to the display 106 and HMD 107. 

The aforementioned processes are repeated until 
the AR game ends . 



With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, an objective viewpoint AR video can be 
presented to a third party other than the player 101 in 
the AR game using the video see-through HMD 107. 
[ Second Embodiment ] 

In the first embodiment, the HMD 107 is of video 
see-through type. However, if the HMD 107 is of 
optical see-through type, the player 101 can still play 
the AR game . 

Fig. 13 shows an optical see-through HMD 1301. 
Note that Fig. 13 schematically illustrates the HMD 
1301, and the present invention is not limited to the 
size and shape shown in Fig. 13. 

Reference numeral 1301 denotes an optical 
see-through HMD; and 1302, the eye of the player 101. 

On the display screen 501, only videos of the 
virtual objects 102 (subjective viewpoint virtual 
videos) are displayed. On the other hand, a video of a 
real space is seen behind the display screen 501 when 
viewed from the position of the eye 1302. Hence, the 
player can review videos of the virtual objects 102 and 
the real space seen behind the display screen 501 when 
viewed from the position of the eye 1302 to overlap 
each other by observing the display screen 501. 



Fig. 14 is a block diagram showing the flow of 
processes of this embodiment. 

A video output from the AR game apparatus 104 to 
the HMD 1301 is a subjective viewpoint virtual video 
5 alone, as described above. Hence, the HMD 1301 and 
subjective viewpoint video generator 212 are 
electrically connected via a cable, and a subjective 
viewpoint virtual video which is generated by the 

Q 

«9 subjective viewpoint video generator 212 on the basis 

EH 

Ml 10 of the position and posture of the HMD 1301 is sent 

from the subjective viewpoint video generator 212 to 
hi the HMD 1301. Note that this embodiment has no 

p subjective viewpoint video composition unit 213 in the 

fh first embodiment. 

m 

& 15 Also, the method of generating an objective 

w viewpoint AR video is the same as that which has been 

explained in the first embodiment. 

The internal block diagram of the AR game 
apparatus 104 in this embodiment is substantially the 
20 same as that shown in Fig. 7, except that the objective 
viewpoint video composition unit 213 is removed from 
the arrangement shown in Fig. 7. 

In the flow chart of this embodiment, steps S804, 
S805, and S806 in Fig. 8 are rewritten as follows. 
25 More specifically, upon generating a subjective 

viewpoint AR video (only videos of the virtual objects 
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102 based on the position and posture of the HMD 1301 
in this embodiment) to be displayed on the HMD 1301, 
steps S804 and S805 are skipped. That is, in step S804 
the objective viewpoint video composition unit 203 
receives an actually sensed video sensed by the camera 
103. In step S805, the objective viewpoint video 
composition unit 203 generates an objective viewpoint 
AR video. In step S806, the objective viewpoint video 
composition unit 203 outputs the objective viewpoint AR 
video to the display 106, and the subjective viewpoint 
video generator 212 outputs a subjective viewpoint 
virtual video to the HMD 1301. 

The flow chart obtained by modifying the contents 
of Fig. 8 as described above is that in this embodiment, 
and this embodiment is controlled by a program code 
according to this modified flow chart. 

With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, an objective viewpoint AR video can be 
presented to a third party other than the player 101 in 
the AR game using the optical see-through HMD 1301. 
[Third Embodiment] 

In the first and second embodiments, the camera 
parameters of the camera 103 are fixed. That is, an 
objective viewpoint AR video is generated based on the 
camera parameters of the camera 103 which is. fixed in 



position. The camera parameter data of the camera 103 
are stored as permanent values in the RAM 703 of the AR 
game apparatus 104. 

A case will be examined below wherein the 
5 viewpoint position, posture, and zooming ratio of the 
camera 103 are changed in real time to those that the 
player or a third party other than the player 
101 (watcher or operator) wants. That is, a case will 
be examined below wherein the camera parameters of the 
lijj 10 camera 103 are changed in real time. Note that this 

4; embodiment uses a video see-through HMD as in the first 

W embodiment. However, the HMD that can be used in this 

D embodiment is not limited to the video see-through type, 

■32- 

p but an optical see-through HMD may be used, as can be 

Q 15 seen from the description of the second embodiment and 

O 

this embodiment. 

When the position, posture, and zooming ratio of 

the camera 103 are to be changed in real time, a 

measurement means as a means for measuring the position, 
20 posture, and zooming ratio of the camera 103 must be 

added to the first embodiment. 

Fig. 4 is a block diagram showing the flow of 

processes in this embodiment. The flow of processes in 

this embodiment will be described below using Fig. 4. 
25 Like in the first embodiment, in this embodiment 

the camera parameter measurement /management unit 204 



- 26 - 



measures and manages the camera parameters of the HMD 
107, and holds some internal parameters of the camera 
103 as known information. Unlike in the first 
embodiment, in this embodiment the camera parameter 
measurement /management unit 204 controls a sensor (not 
shown) attached to the camera 103 to measure the 
position, posture, and zooming ratio of the camera 103. 

When the player 101 has changed the position, 
posture, and zooming ratio of the camera 103 via an 
interface (not shown) , the camera parameter 
measurement /management unit 204 measures the position, 
posture, and zooming ratio of the camera 103. The 
measurement result data (the position, posture, and 
zooming ratio data of the camera 103) by the camera 
parameter measurement /management unit 204, and the 
known internal parameters are output to the objective 
viewpoint video generator 202. The objective viewpoint 
video generator 202 generates an objective viewpoint 
virtual video on the basis of the input camera 
parameters of the camera 103 in the same manner as in 
the process in the first embodiment. Other processes 
are the same as those in the first embodiment. 

A control program code of the interface that 
changes the position, posture, and zooming ratio of the 
camera 103, and a control program code of the camera 



parameter measurement /management unit 204 are stored in 
the RAM 703 of the AR game apparatus 104. 

The flow chart of this embodiment is the same as 
that shown in Fig. 8. However, in this embodiment the 
5 camera parameter measurement /management unit 204 

updates the information of the viewpoint position and 
posture of the HMD .107 and the information of the 
viewpoint position, posture, and the zooming ratio of 
the camera 103 in step S801. 

10 Internal parameter (zooming ratio) is fixed, 

position and posture may be variable. With the 
aforementioned arrangement of the apparatus, augmented 
reality presentation method, and program code, an 
objective viewpoint AR video can be generated based on 

15 data from the camera whose position and posture change. 
[ Fourth Embodiment ] 

In the first to third embodiments, only one 
camera is set. Alternatively, when a plurality of 
cameras are set and the camera to be used is switched, 

20 objective viewpoint AR videos from a plurality of 

positions and postures can be displayed on the display 
106. 

Fig. 9 shows a camera system in this embodiment. 
The camera system will be explained below using Fig. 9. 
25 Note that this embodiment sets three cameras. 
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Reference numerals 901a, 901b, and 901c denote 
cameras (cameras 901a, 901b, and 901c) , the camera 
parameters (positions, postures, field angles, and the 
like) of which are fixed- The functions of these 
cameras are the same as those in the first and second 
embodiments. The camera parameters of these three 
cameras are set, as shown in Fig. 12, and these three 
sets of data are stored in the RAM 703 of the AR game 
apparatus 104 and are managed by the camera parameter 
measurement/management unit 204. 

Note that reference numerals 1201, 1202, and 1203 
in Fig. 12 denote (selection information) tags 
corresponding to the cameras 901a, 901b, and 901c, 
which are used to select the camera to be used. 

Reference numeral 902 denotes a camera switching 
device, which has a function of a switcher that sends 
only a signal (actually sensed video of a real space 
sensed by the selected camera) from the camera selected 
by a selector 903 to the video composition unit 203 via 
the switching device 902. 

Reference numeral 903 denotes a selector which 
has the aforementioned function, and has three buttons 
(buttons A, B, and C) corresponding to the three 
cameras. When the user selectively presses one of 
these buttons, he or she can select the camera to be 
used. The selector 903 outputs selection information 



indicating the camera selected to the switching device 
902 and camera parameter measurement /management unit 
204. Note that the cameras 901a, 901b, and 901c are 
respectively selected by pressing buttons A, B, and C 
in Fig. 9 as the camera to be used. 

The camera parameter measurement /management unit 
204 selects the tag (one of the tags 1201, 1202, and 
1203) of the camera mentioned above in accordance with 
the camera selection information input from the 
selector 903, and outputs the camera parameters of the 
selected camera to the objective viewpoint video 
generator 202. For example, when the camera 901b is 
used, the camera parameters of the camera 901b held by 
the tag 1202 are output to the objective viewpoint 
video generator 202 upon depression of button B. 

Fig. 10 is a flow chart showing the process of 
this embodiment. 

It is checked in step S1001 if one of the buttons 
of the selector 903 has been pressed. Note that this 
process is repeated until one of the buttons is pressed. 

In steps S1002, S1003, and S1004, the button 
pressed is determined. Assuming that button B has been 
pressed, the flow advances from step S1003 to step 
S1006. 

In steps S1005, S1006, and S1007, an actually 
sensed video sensed by the camera corresponding to the 



pressed button is sent to the objective viewpoint video 
composition unit 203 via the switching device 902. 
Also, the selection information of the camera 
corresponding to the pressed button is sent to the 
5 camera parameter measurement /management unit 204. 

A program code according to the aforementioned 
flow chart is stored in the internal memory (not shown) 
of the switching device 902. 

Camera switching may be automatically performed 

10 in accordance with the progress of a game. In this 

case, the selector 903 is omitted, and the game state 
manager 201 outputs camera selection information to the 
camera switching device 902 and the camera parameter 
measurement /management unit 204. Camera switching can 

15 employ a method of switching cameras every 

predetermined time interval or a method of switching 
cameras in accordance with the progress of a scenario 
by presetting a camera having an appropriate camera 
angle every time the scenario progresses. 

20 In this embodiment, the camera parameters of the 

three cameras are fixed. Alternatively, when the 
camera parameters of these cameras change in real time, 
the aforementioned arrangement of the apparatus, 
augmented reality presentation method, and program code 

25 can be applied. In this case, such application can be 
implemented by connecting the camera system of this 
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embodiment to the AR game apparatus 104 in place of the 
camera 103 in the third embodiment. 

With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, a plurality of objective viewpoint AR 
videos sensed by a plurality of cameras can be 
displayed on the display 106. 
[Fifth Embodiment] 

In the first to fourth embodiments, an objective 
viewpoint AR video is presented to a third party other 
than the player via the display 106 by outputting it to 
the display 106. Alternatively, the objective 
viewpoint AR video may be presented to the player 101. 
That is, a display area 601 shown in Fig. 6 is assured 
on the display screen 501 of the HMD (which can be of 
either video or optical see-through type) that the 
player 101 wears, and the objective viewpoint AR video 
is displayed there. In Fig. 6, a video displayed on 
this display screen 501 will be referred to as an 
augmented video hereinafter. 

In order to generate this augmented video, a 
program code for setting the display area 601 on the 
display screen 501, and writing an objective viewpoint 
AR video on that display area 601 is stored in the RAM 
703 of the AR game apparatus 104 in addition to the 
program code according to the flow chart shown in 



Fig. 8. As a result, by executing this program code, 
the augmented video can be displayed on the display 
screen 501. 

Fig. 11 is a flow chart of the program code for 
writing the objective viewpoint AR video on the display 
area 601- mentioned above. 

It is checked in step S1101 if the display area 
601 is assured. Selection as to whether or not the 
display area 601 is assured can be implemented by 
providing this selection switch to an operation device 
(not shown) used when the player 101 plays the AR game. 
Or such selection can be implemented by inputting a 
command indicating whether or not the display area 601 
is assured from the console 705. 

In step S1102, the display position of the 
display area 601 is input. The display position is 
input from the console 705. Alternatively, the player 
101 may input the display position using the 
aforementioned operation device. 

In step S1103, the size of the display area 601 
is input. The size is input from the console 705. 
Alternatively, the player 101 may input the size using 
the aforementioned operation device. 

In step S1104, the display area 601, the setups 
of which have been determined in steps S1102 and S1103, 
is assured on the display screen 501. 



In step S1105, the objective viewpoint AR video 
generated by the objective viewpoint video composition 
unit 203 is rendered on the display area 601. As a 
consequence, the augmented video can be generated. 

Note that this augmented video may be output to 
the display 106. 

Selection of whether the display area 601 is set 
and the display position and size of the display area 
601 may be automatically set/changed not by the player 
but in accordance with the progress of the game. In 
this case, the game state manager 201 determines these 
parameters . 

The display area 601 may be the entire area of 
the display screen 501. 

With the aforementioned arrangement of the 
apparatus, augmented reality presentation method, and 
program code, both the subjective and objective 
viewpoint AR videos can be presented to the player. 
[Sixth Embodiment] 

A plurality of players 101 may join the AR game. 
In this case, subjective viewpoint AR videos from 
individual subjective viewpoints must be provided to 
the individual players. Fig. 15 shows the internal 
arrangement of the AR game apparatus for providing 
subjective viewpoint AR videos from the players' 
subjective viewpoints to the individual players. Note 



that the camera 103 is fixed in position, and the 
processes that pertain to the camera 103 and display 
are the same as those in the first embodiment . 

Fig. 15 shows the internal arrangement for three 
players. Hence, the AR game apparatus 104 comprises 
HMDs 107A, 107B, and 107C, subjective viewpoint video 
composition units 213A, 213B, and 213C, and subjective 
viewpoint video generators 212A, 212B, and 212C in 
correspondence with three players a, b, and c. Three 
players a, b, and c respectively wear the HMDs 107A, 
107B, and 107C. This embodiment uses a video 
see-through HMD, but an optical see-through HMD may be 
used. In this case, the subjective viewpoint video 
composition units for the three HMDs can be omitted. 

When the AR game starts, the HMDs, subjective 
viewpoint video generators, and subjective viewpoint 
video composition units execute the same processes as 
those described in the first embodiment, and subjective 
viewpoint AR videos generated for the individual 
players are output to the HMDs 107A, 107B, and 107C 
that the players wear. 

Note that the aforementioned arrangement of the 
apparatus and augmented reality presentation method are 
not limited to three players, as is obvious from the 
above description . 



With the aforementioned arrangement of the 
apparatus and augmented reality presentation method, 
subjective viewpoint AR videos can be provided to a 
plurality of players. 
5 [Seventh Embodiment] 

In the fourth embodiment, a plurality of cameras 
103 are set, but only one display 106 for displaying an 
objective viewpoint AR video is used, 
yp However, the number of displays 106 is not 

Ijl 10 limited to one, but a plurality of displays 106 may be 

used. That is, when objective viewpoint AR videos from 

3 3 ~ 

n the respective cameras 103 are separately displayed on 

m a plurality of displays 106, the objective viewpoint AR 

videos from all the cameras 103 can be presented to a 

s 5 

ffi 15 third party other than the player 101 using a plurality 

C9 of displays 106. 

In this embodiment, a case will be described 
below wherein a plurality of cameras 103 and a 
plurality of displays 106 are set. Fig. 16 shows the 
20 internal arrangement of the AR game apparatus 104 for 
displaying objective viewpoint AR videos from a 
plurality of cameras 103 on a plurality of displays 106. 

In Fig. 16, three cameras 103A, 103B, and 103C 
are set, and displays 106A, 106B, and 106C for 
25 respectively displaying objective viewpoint AR videos 
from the cameras 103A, - 103B, and 103C are set. Also, 
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the AR game apparatus 104 comprises objective viewpoint 
video generators 202A, 202B, and 202C, and objective 
viewpoint video composition units 203A, 203B, and 203C 
corresponding to the displays 106A, 106B, and 106C so 
as to generate objective viewpoint AR videos 
corresponding to the displays 106A, 106B, and 106C. 

When the AR game starts, the cameras, objective 
viewpoint video generators, and objective viewpoint 
video composition units execute the same processes as 
those in the first embodiment, and objective viewpoint 
AR videos from the cameras 103A to 103C are displayed 
on the displays 106A*to 106C corresponding to these 
cameras 103A to 103C. 

Note that the aforementioned arrangement of the 
apparatus and augmented reality presentation method are 
not limited to three displays 106, as is obvious from 
the aforementioned description. 

With the aforementioned arrangement of the 
apparatus and augmented reality presentation method, 
all objective viewpoint AR videos from a plurality of 
cameras 103 can be presented to a third party other 
than the player 101 using a plurality of displays 106. 
[ Eighth Embodiment ] 

An apparatus that offers an AR video to the 
player 101 may be other than the game apparatus. That 
is, an AR apparatus which has substantially the same 



internal arrangement as that of the AR game apparatus 
104 and stores a program code for implementing shopping 
in an AR space in place of a game program code may be 
used in place of the AR game apparatus 104. As a 
5 result, the player 101 can browse among presented 
products in shopping by observing a subjective 
viewpoint AR video from the AR apparatus, and that 
state is displayed on the display 106 as an objective 

if* 

X viewpoint AR video. 

s as 

10 Note that these subjective and objective 

J: viewpoint AR videos can be generated in the same manner 

y * 

W as in the above embodiments. 

? 

G [Ninth Embodiment] 

Q When the camera parameters such as the position, 

fri 

p 15 posture, zoom ratio, and the like of the camera change, 

Q 

they may be estimated from an actually sensed video 
input from the camera 103 in place of the sensor. In 
this case, the camera parameter measurement /management 
unit 204 receives an actually sensed video, and 

20 estimates camera parameters by a known method. In 
consideration of estimation errors, the camera 
parameters measured by the sensor and those estimated 
by the camera parameter measurement /management unit 204 
may be averaged to determine camera parameters to be 

25 used finally. 

[Tenth Embodiment] 
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The generated objective viewpoint AR video can be 
not only displayed on the display 106 but also printed 
on a medium such as paper or a film. In this case, a 
print controller 301 (print control apparatus 301) and 
5 a printer 302 are added to the arrangement shown in 
each of the first to seventh embodiments. The 10th 
embodiment will be described with reference to Fig. 17 
by exemplifying the arrangement of the first embodiment. 

ifTi 

In the 10th embodiment, an objective viewpoint AR 

y s 

10 video output from an objective viewpoint video 
composition unit 203 is output to the printer 
W controller 301 as well as the display 106. The print 

O controller 301 stores the objective viewpoint AR video 

•as 

Q input at this time point and outputs the stored image 

m 

Q 15 to the printer 302 in accordance with a command input 

~~ from a console 705. The printer 302 prints the input 

image on paper. 

As in the seventh embodiment, when a plurality of 
objective viewpoint video composition units 203 are 
20 used, the print controller 301 receives objective 
viewpoint AR videos respectively output from the 
objective viewpoint video composition units 203. In 
this case, in accordance with a command input from the 
console 705 the print controller 301 selects an 
25 objective viewpoint AR video having a specific 

viewpoint, stores the objective viewpoint AR video 
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input at this time point, and outputs the stored image 
to the printer 302. It is possible to print the 
subjective viewpoint AR video of each embodiment 
described above in accordance with the same processing 
5 as described above. In this case, a subjective 
viewpoint video composition unit 213 outputs the 
subjective viewpoint AR video to an HMD 107 and at the 
same time to the print controller 301. 

It is also possible to simultaneously print 

10 objective viewpoint AR videos viewed from a plurality 
of viewpoint positions, objective viewpoint AR videos 
viewed from the same viewpoint at different times, or 
subjective viewpoint AR videos of the respective 
players on one paper sheet. In this case, the print 

15 controller 301 selects a viewpoint in accordance with a 
command input from the console 705 and stores the AR 
video viewed from the selected viewpoint. The print 
controller 301 determines a layout for printing a 
plurality of stored images on one paper sheet in 

20 accordance with a command input from the console 705 
and outputs them to the printer 302. 

The command may be automatically input to the 
print controller 301 in accordance with the progress of 
a game without using the console 705. In this case, a 

25 game state manager 201 transmits a command to the print 
controller 301. The command may be transmitted at a 
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fixed timing preset in accordance with the progress of 
the game or a dynamic timing corresponding to the 
progress of the game such that the distance between the 
player and a virtual object 102 becomes a predetermined 
interval or less. 

The camera 103 (or camera 901) can be located at 
an arbitrary position. The third party can easily 
grasp the state of the game when the camera is located 
at a position where the entire AR space can be observed 
from the bird-view position or a position where the 
upper half image (so-called bust shot) of the player 
can be sensed from the front. 

As described above, according to the present 
invention, a video of the AR game viewed from the 
subjective viewpoint of the player can be generated, 
and, simultaneously, a video of the AR game viewed from 
an objective viewpoint to see the overall view of the 
game field or a video of the AR game viewed from a 
viewpoint the watcher wants can be generated. 

As many apparently widely different embodiments 
of the present invention can be made without departing 
from the spirit and scope thereof, it is to be 
understood that the invention is not limited to the 
specific embodiments thereof except as defined in the 
appended claims. 



